The test of software products (to verify their correctness, completeness and quality) is a very critical process. The execution of a test process generally requires performing selected operations and then verifying whether the software product behaves as expected. For this purpose, each tester interacts with the software product by means of a corresponding interface, which may be of the command line type or of the graphical type; particularly, in a Command Line Interface (CLI) the commands with respective arguments are provided in a textual format (from a keyboard or a script file), while in a Graphical User Interface (GUI) the commands are issued by manipulating images on a monitor with a pointer (of a mouse). The CLI is very powerful and allows performing certain operations more quickly; generally, the CLI is used by expert end-users (such as in programming environments). Conversely, the GUI is user-friendly and simpler; therefore, the GUI is well suited for less skilled end-users (such as in secretarial environments).
The design of the test process generally starts with the definition of its coverage. A technique commonly used for this purpose consists of creating a scenarios matrix; this matrix lists a series of test scenarios, each one for verifying a selected feature of the software product. Considering in particular the CLI, the scenarios matrix includes the invocation of the commands to be tested with different values of their arguments. For example, the commands are exercised with all the required arguments, with some arguments missing, or with the arguments in the right/wrong order; likewise, the arguments are selected in the allowable ranges, at their boundaries, outside the allowable ranges, with right/wrong formats, and the like.
Starting from the scenarios matrix, a series of corresponding test cases is then defined. Each test case indicates the execution of a selected command with the desired values of the arguments, followed by the verification of its expected result; for example, this requires specifying for each test scenario an output message (such as a return code and/or an explanation text), which should be received from the software product. The test cases are described in prose in a document to be used for executing them manually. Alternatively, it is also possible to program an automation tool accordingly so as to cause the automatic execution of the test cases.
In any case, the management of the test process is substantially a manual task. Indeed, all the activities relating to the design of the scenarios matrix and of the test cases remain in charge of a test manager. Very often, the execution of the test cases as well is performed manually; however, even when an automation tool is available, its programming requires a heavy human intervention.
The problem is particular acute in large software products. Moreover, the same operations must be repeated at any new version, release or service level of the software product. In this respect, it should be noted that the test of the software product may be required at different levels of accuracy, from a basic level (when only the building of the software product has to be verified) to a high level (when each component thereof has to be verified).
In any case, the work required to design the test process is time consuming, repetitive and tedious; likewise, its execution is a long and boring activity. Moreover, the quality of the test process strongly depends on the skill of the test manager; therefore, the obtained results are not repeatable and prone to errors. The same document (which is intended to describe the test cases) is not well suited to convey and/or share the desired information, and does not allow an objective analysis of the test process.
All of the above adversely affects the cost and the quality of the test process; this hinders its extensive application, with a negative repercussion on the reliability of the software products.